Increased intelligence is a myth (so far)

On one hand, intelligence testing is one of the great successes of psychology (Hunt, 2011). Intelligence test scores predict many real world phenomena and have many well-validated practical uses (Gottfredson, 1997; Deary et al., 2010). Intelligence test scores also correlate to structural and functional brain parameters assessed with neuroimaging (Haier et al., 1988; Jung and Haier, 2007; Deary et al., 2010; Penke et al., 2012; Colom et al., 2013a) and to genes (Posthuma et al., 2002; Hulshoff Pol et al., 2006; Chiang et al., 2009, 2012; Stein et al., 2012). On the other hand, intelligence test scores are often misunderstood and can be misused. This paper focuses on a basic misunderstanding that permeates many of the recent reports of increased intelligence following short-term cognitive training. Several of these reports have been published in prominent journals and received wide public attention (Jaeggi et al., 2008, 2011; Mackey et al., 2011). The basic misunderstanding is assuming that intelligence test scores are units of measurement like inches or liters or grams. They are not. Inches, liters and grams are ratio scales where zero means zero and 100 units are twice 50 units. Intelligence test scores estimate a construct using interval scales and have meaning only relative to other people of the same age and sex. People with high scores generally do better on a broad range of mental ability tests, but someone with an IQ score of 130 is not 30% smarter then someone with an IQ score of 100. A score of 130 puts the person in the highest 2% of the population whereas a score of 100 is at the 50th percentile. A change from an IQ score from 100 to 103 is not the same as a change from 133 to 136. This makes simple interpretation of intelligence test score changes impossible. Most recent studies that have claimed increases in intelligence after a cognitive training intervention rely on comparing an intelligence test score before the intervention to a second score after the intervention. If there is an average change score increase for the training group that is statistically significant (using a dependent t-test or similar statistical test), this is treated as evidence that intelligence has increased. This reasoning is correct if one is measuring ratio scales like inches, liters or grams before and after some intervention (assuming suitable and reliable instruments like rulers to avoid erroneous Cold Fusion-like conclusions that apparently were based on faulty heat measurement); it is not correct for intelligence test scores on interval scales that only estimate a relative rank order rather than measure the construct of intelligence. Even though the estimate has considerable predictive value and correlates to brain and genetic measures, it is not a measurement in the same way we measure distance, liquid, or weight even if individual change scores are used in a pre-post design. SAT scores, for example, are highly correlated to intelligence test scores (Frey and Detterman, 2004). Imagine a student takes the SATs when quite ill. The scores likely are a bad estimate of the student’s ability. If the student retakes the test sometime later when well, does an increase in score mean the student’s intelligence has increased, or that the newer score is now just a better estimate? The same is true for score changes following SAT preparatory courses. Many colleges and universities allow applicants to submit multiple SAT scores and the highest score typically carries the most weight; there are many spurious reasons for low scores but far fewer for high scores. Change scores from lowest to highest carry little if any weight. By contrast, change in a person’s weight after some intervention is unambiguous. In studies of the effect of cognitive training on intelligence, it is also important to understand that all intelligence test scores include a certain amount of imprecision or error. This is called the standard error of measurement and can be quantified as an estimate of a “true” score based on observed scores. The standard error of measuring inches or liters is usually zero assuming you have perfectly reliable, standard measurement devices. Intelligence tests generally show high test-retest reliability but they also have a standard error, and the standard error is often larger for higher scores than for lower scores. Any intelligence test score change after an intervention needs to be considered relative to the standard error of the test. Studies that use a single test to estimate intelligence before and after an intervention are using less reliable and more variable scores (bigger standard errors) than studies that combine scores from a battery of tests. Change scores are never easy to interpret and require sophisticated statistical methods and research designs with appropriate control groups. If you try a training intervention in individuals all of whom have pre-intervention scores below the population mean, for example, re-testing with or without any intervention, may result in higher scores due to the statistical phenomenon of regression to the mean, or due to simple test practice, especially if equivalent alternative forms of